Many software vendors now provide facilities that enable their customers to extend or customize their software applications by integrating external routines designed and developed by the customers themselves or by third-party software developers. For performance reasons, these external routines are usually coded in a third-generation language (3GL) such as C or C++, compiled into native machine language instructions specific to a particular platform (processor type and operating system), and archived into a library. The vendor's base software is then linked with the library to integrate the external routines contained in the library with the base software. In recent years, dynamic link libraries (DLLs) have become popular, whereby external routines in a DLL are linked into the base application at start-up or as needed during the execution.
For example, one such software system is disclosed in the commonly assigned, co-pending U.S. patent application Ser. No. 08/880,326 entitled "Apparatus and Method for Calling External Routines in a Database System" filed on Jun. 23, 1997 by Jacco Draaijer, Roger Bodamer, and Eric Voss, the contents of which are hereby incorporated by reference. Therein, users have the capability of writing external procedures in a 3GL and compiling them to be called from within a database system. Accordingly, a database user can use a database implementation language, such as SQL (Structured Query Language) or a procedural language extension such as PL/SQL, available from Oracle Corp. of Redwood Shores, Calif., to call the foreign 3GL external procedures as if they were internal procedures or functions. The external functions are archived into a shared library such as a DLL and registered with the base software. At runtime, facilities associated with the base software dynamically load the shared library and invoke routines therein directly or indirectly via an external agent. Thus, users are able to invoke, from within the base database application, external software they have developed themselves in a 3GL language.
It is, of course, inevitable that at least some software applications contain runtime faults, which can crash and prevent the application from completing. For example, de-referencing an invalid pointer on some platforms causes a segmentation fault, which immediately halts execution of a program. On other platforms, this fault results in an invalid value being fetched from memory, thereby introducing a subtle error that may not become apparent until much later. A difficulty that arises with integrated software, however, is determining whether the fault lies with the external software or with the base software, because the fault typically does not become apparent until after the external software and the base software have been integrated. Thus, the job of debugging software is rendered very difficult (if not impossible) when the software developer is unable to even identify which software module actually contains the fault. This difficulty is exasperated because the developer of the external routine almost never has access to the base software source code and the base software programmer is unfamiliar with the implementation of the external routines. Thus, neither expert is proficient with the other's code, thereby leading one developer to suspect the other's code to be at fault.
A common conventional approach for determining the location of a crash-inducing software fault is to perform a detailed examination of a "core dump," produced on some platforms when a process is aborted by certain kinds of internal errors, such as a segmentation violation (a kind of memory error on virtual memory systems) or execution of an illegal instruction (typically caused, e.g., by a corrupted return address on the stack). A core dump contains the contents of the internal memory used by a process and typically includes the values of allocated virtual memory, user registers, and system registers such as a status register and program counter, often displayed as a raw collection of hexadecimal or octal numbers. A core dump is usually examined by an expert programmer with a symbolic debugger in conjunction with the source code for the external routines and the base software. Although the location of some faults might be ascertained by asking the symbolic debugger to trace the program stack, which identifies the routine in which the run-time fault manifested, other errors, such as a corrupted stack, are much more difficult to isolate to a specific module.
Examining a core dump, however, involves significant logistical difficulties, because the core dump is a very complicated and extremely large file. For example, the complexity of core dumps makes it difficult for a support person such as a help desk analyst to assist the customer over the telephone. Because core dumps commonly contain many megabytes of storage, it can take a significant amount of time to download the core dump from the customer's platform to a test computer on a compatible platform for debugging. In addition, symbolic debuggers typically require a copy of the external DLL and, preferably, the customer's source code, which the customer may be reluctant to disclose. Furthermore, not all platforms generate core dumps.